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The  Problem 

Quite  often,  software/system  improvements  are  made  without 
measurement  either  before  or  after  the  change  was  introduced. 

Therefore,  how  do  we  know  if  the  outcome  was  better  or  worse 
than  the  original  situation? 

In  these  cases,  interpretation  is  based  on  opinion  and 
impressions  -  but  there  is  a  lack  of  data  to  back  it  up! 

Therefore,  there  is  less  confidence  in  the  results  of  the 
innovation  -  interpretation  is  problematic  and  there’s  a  risk  that 
consensus  about  the  results  are  not  achieved. 


How  do  we  know  if  the  change  worked? 
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Need  to  Validate  That  Changes  Are 
Effective 


How  do  we  know  it  worked? 


,rNe* 


As-ls  State 

) y  Or  not? 

i  ne  New  as-is  otate 
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Typical  Approaches  to  Evaluating 
Improvement 

Typical  approaches  that  fail 


Approach  #1 

Before  the  change  Change  introduced  After  the  Change 

X 

Approach  #2 

X  o 

Approach  #3 

0  X 

X  represents  the  introduction  of  a  change 
O  represents  a  measurable  observation 
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How  Do  You  Know  It  Really  Worked? 


Typical  approaches  that  fail 


Before  the  change  Change  introduced  After  the  Change 


In  all  three  approaches,  there  is  no  way  to  tell  if  the  outcome  from 
the  change  was  better  or  worse  than  the  original  situation. 


How  do  you  know 
the  change  worked? 
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Scientific  Methods  Do  Exist 


Research  designs  do  exist  for  proper  interpretation  of 
results  after  innovations  are  introduced  ...  but  they  are 
rarely  applied! 


Before  the  change 

Change  introduced 

After  the  Change 

°i 

:  § 

X 

°2 

1  ■  IPS 

■ 

1 

As-ls  State  transition  To-Be  State 
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A  Structured  Approach  to  a  Pilot  Study 


1 

Plan  and  design  the  pilot  study 

2 

Train  personnel  to  accomplish  change 

3 

Support  and  monitor  pilot  study 

4 

Evaluate  pilot  results 

5 

Make  recommendations  &  improve 
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A  Structured  Approach  to  a  Pilot  Study 


1  |  Plan  and  design  the  pilot  study  | 

2  I  Train  personnel  to  accomplish  change 

3  I  Support  and  monitor  pilot  study  HH 

4  I  Evaluate  pilot  results 

5  1  Make  recommendations  &  improve  I 


•  How  will  you  measure 
success? 


•  Where  will  you 
conduct  the  pilot 
study? 

•  Designing  your 
approach  using 
scientific  principles 

•  Writing  down  your 
plan 
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Define  the  Problem 

A  problem  that  is  clearly  defined  is  half-solved. 

Defining  the  problem  means  identifying  a  gap  between  some 
desired  situation  and  the  current  situation. 

An  important  challenge  is  for  the  improvement  team  to 
collect  and  use  valid  information  to  define  the  current 
situation  instead  of  assuming  that  it  already  has  the 
necessary  valid  information. 

A  problem  statement  implies  no  particular  solutions  and  no 
potential  causes. 

A  good  problem  statement  states  only  the  current  and 
desired  situation. 
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A  Structured  Approach  to  a  Pilot  Study 


1  |  Plan  and  design  the  pilot  study  | 

2  Train  personnel  to  accomplish  change 

3  I  Support  and  monitor  pilot  study  hBH 

4  I  Evaluate  pilot  results 

5  fj  Make  recommendations  &  improve  I 


Subtopics 


*  Define  the  problem 


•  Where  will  you 
conduct  the  pilot 
study? 


•  Designing  your 
approach  using 
scientific  principles 

•  Writing  down  your 
plan 
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Develop  Pilot  Study  Success  Criteria 


You  will  want  to  know: 

Did  the  solution  component  generate  the  outcome  that  it  was 
intended  to  achieve? 

•  What  are  you  hoping  for  in  terms  of  performance  change 
when  using  the  solution  component? 

•  Try  to  define  performance  standards  that  will  help  you 
determine  this  explicitly.  Are  there  any  historical  data  that  can 
be  used  to  baseline  the  status  quo? 

Did  the  users  experience  difficulty  in  its  use? 

•  What  are  your  expectations  in  terms  of  the  solution 
component’s  impact  on  changing  people’s  attitudes  and 
behaviors? 

•  Try  to  define  qualitative  measures  that  will  provide  objective 
©2ooarssessmentJofrjob  improvement  for  the  users  of  the 
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A  Structured  Approach  to 


1  |  Plan  and  design  the  pilot  study  | 

2  I  Train  personnel  to  accomplish  change 

3  I  Support  and  monitor  pilot  study 

4  I  Evaluate  pilot  results 

5  i  Make  recommendations  &  improve  I 


Pilot  Study 


Subtopics 

*  Define  the  problem 

*  How  will  you  measure 
success? 


•  Designing  your 
approach  using 
scientific  principles 


•  Writing  down  your 
plan 
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Will  we  be  Able  to  Generalize  Our  Findings? 


Is  there  anything  we  can  do  to  increase  the  probability  that 
we  can  generalize  the  pilot  study  results  to  the  larger 
population? 

•  Is  this  a  typical  program/project  within  the 
organization? 

•  Is  the  experience  and  skill  level  of  the  pilot  study 
personnel  typical  of  what  one  would  find  in  other 
programs/projects  in  the  organization? 

•  Are  there  factors  beyond  our  control  that  can  confound 
or  influence  the  cause-d 
change  we  are  trying  to 


Making  smart  decisions  about  where  (in 
your  organization)  to  conduct  a  pilot 
study  improves  confidence  with 
generalizing  the  solution  to  other  parts 
of  the  organization. 
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Understanding  the  Pilot  Environment 

During  pilot  study  planning,  you  can  mitigate  the  risk  of  misinterpreting  or 
over-interpreting  your  eventual  results  if  you  can  identify  and  characterize 
the  impact  of  potential  influences. 

•  How  similar  is  the  project/program  environment  of  the  candidate  pilot 
project  to  other  projects  in  the  organization  (size,  domain,  etc.)? 

•  Will  participants  in  the  pilot  study  embrace  the  proposed  change  (that 
is  being  considered)  or  resist  adopting  it? 

•  What  are  the  adoption  characteristics  of  the  pilot  project  manager  and 
the  project  staff  in  general? 

•  Given  the  adoption  characteristics  of  the  pilot  participants,  how  much 
refinement  (of  the  solution  component)  and  support  will  be  necessary 
to  test  the  potential  effectiveness  of  change? 

•  How  supportive  is  the  project/program  manager  to  the  change  that  will 
be  piloted?  Are  they  enthusiastic  about  the  idea  of  serving  as  a  pilot? 

•  What  kinds  of  pressure  is  the  project/program  already  under? 

-  Difficult  schedule  constraints? 

-  New  product  or  domain  area? 
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Project  Categories  Example 


1  Project  Type 

1  Description 

Critical 

•  on-time  release  is  imperative 

•  20-30  individuals  on  project  staff 

•  using  new  object-oriented  technology 

Product 

enhancement 

•  5-10  individuals  on  project  staff 

•  improvements  to  baseline  products 

Maintenance 

•  <  5  individuals  on  project  staff 

•  correction  of  bugs  reported  by  users 

| 

Emergency 

•  5-20  individuals  on  project  staff 
(dependent  on  need) 

•  project  formed  to  address 
unanticipated  government  mandated 
order 
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Can  We  Generalize  the  Pilot  Results? 
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A  Structured  Approach  to  a  Pilot  Study 


1  ■  Plan  and  design  the  pilot  study 


Subtopics 

•  Define  the  problem 


2  ■  Train  personnel  to  accomplish  change 

3  I  Support  and  monitor  pilot  study 


*  How  will  you  measure 
success? 

•  Where  will  you 
conduct  the  pilot 
study? 


| 


5  ■  Make  recommendations  &  improve 


•  Writing  down  your 
plan 
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Approaches  to  Validation 


In  the  scientific  and  manufacturing  world,  improvements  or 
innovations  are  validated  using  a  rigorous  statistical  approach 
known  as  design  of  experiments  (DOE) 

•  Extraneous  variables  that  might  impact  the  result  you’re 
looking  at  can  be  held  steadied  or  controlled 


•  The  experimental  design  can  employ  techniques  such  as 
randomization  and  replication  to  add  clarity  and  confidence 
to  the  assertions  that  are  made  about  the  change 
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The  Human  Side  of  Change 


In  a  human-related,  real-time  program/project  environment,  pilot 
studies  are  not  conducted  in  a  laboratory  under  ideal  controlled 
conditions. 

•  It  is  difficult — if  not  impossible-  to  control  variables 
involving  people. 

•  The  characteristics  of  the  experimental  medium  (i.e.,  the 

people  using  the  solution  component)  may  influence  the 
results  and  “  ‘  "  "  ack  that  you  obtain  after  you 

introduce  t  ables  affecting 


New  innovation 


Introduction  of 
Software  Proce: 


morale  of  Technical  Staff 


performance 


SE  staff 


Resistance 
to  change 
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Scientific  Methods  Do  Exist 


Research  designs  called  quasi-experimentation  do  exist  for 
proper  interpretation  of  results  from  pilot  studies  ...  but 
they  are  rarely  applied! 


Before  the  change 


Change  introduced 


After  the  Change 


& 


- 

Before  the  change 

Group  #1 
Group  #2 


<!»e 


Change  introduced 


After  the  Change 


A  t  test  and  the  analysis  of  covariance  method  are  statistical  methods  that  provide  a 
scientific  basis  for  making  assertions  about  the  results  of  change  effort. 
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A  Structured  Approach  to 


1  |  Plan  and  design  the  pilot  study  | 

2  I  Train  personnel  to  accomplish  change 

3  I  Support  and  monitor  pilot  study 

4  I  Evaluate  pilot  results 

I 


Pilot  Study 


Subtopics 

*  Define  the  problem 

*  How  will  you  measure 
success? 

*  Where  will  you 
conduct  the  pilot 
study? 


•  Designing  your 
approach  using 
scientific  principles 
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Pilot  Implementation  Plan 

The  pilot  implementation  plan  is  developed  collaboratively  with  the 
people  who  will  participate  in  the  pilot  study. 

Pilot  studies  involve  altering  the  environment  (e.g.,  how  they  work)  of 
the  participants. 

Therefore,  changes  must  be  introduced  carefully  so  that  they  don’t 
become  overwhelmed. 

Introducing  too  many  variables  at  one  time  will  make  the  results 
difficult  to  interpret. 

A  plan  is  developed  that  describes  how  solution  component(s)  will  be 
introduced  over  time  into  a  program/project  for  pilot  testing. 

The  plan  provides  a  mechanism  for  setting  the  appropriate 
expectations  with  the  project  partner. 
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What’s  in  the  Plan? 

The  plan  includes 

•  objectives  of  the  pilot  study 

•  success  indicators  and  how  they  are  measured 

•  approach 

•  responsibilities  of  participants 

•  training  activities 

•  description  of  support  mechanisms  (e.g.,  mentoring,  hot-line 
support,  trouble-shooting) 

•  a  description  of  pilot  study  retrospective  activity 

•  a  schedule 

•  risks  and  mitigation  strategies 
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A  Structured  Approach  to  a  Pilot  Study 


Plan  and  design  the  pilot  study 


2  II  Train  personnel  to  accomplish  change 


Support  and  monitor  pilot  study 


Evaluate  pilot  results 


Make  recommendations  &  improve 
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Don’t  Overlook  the  Need  for  Training 

The  type,  style  and  extent  of  training  will  depend  on  the  complexity  of 
the  proposed  change. 

The  pilot  implementation  plan  describes  the  training  activities. 

The  training  should  cover 

•  how  the  new  process,  procedure,  and/or  associated  technology 
is  performed  or  used 

•  how  to  use  the  documentation  that  describes  the  new  feature 

•  how  to  obtain  additional  help  if  there  are  problems 


Ensure  that  feedback 
mechanisms  are  included  as 
part  of  the  training  approach. 
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A  Structured  Approach  to  a  Pilot  Study 


■ _ I 

2  I  Train  personnel  to  accomplish  change 

3  |  Support  and  monitor  pilot  study| 

I _ I 

5  i  K§ 
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Supporting  and  Monitoring  the  Pilot  Effort 

Any  change,  however  well-planned,  can  cause  unanticipated  results. 

That’s  why  we  conduct  a  pilot  test  -  we’re  not  sure  what  to  expect. 

Pilot  personnel  will  need  help  when  problems  are  exposed. 

Support  must  be  provided  during  pilot  testing  so  that  program/project 
personnel  can  obtain  quick  solutions  to  glitches  or  bugs  in  the  new 
process  component(s). 

A  member  of  the  pilot  project  support  team 

•  is  assigned  as  the  primary  point  of  contact  to  provide  guidance 
or  help  when  problems  arise 

•  is  responsible  for  ensuring  that  performance  indicators  are 
tracked  throughout  the  pilot  effort. 

Document  problems  or  issues  as 
they  occur. 
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A  Structured  Approach  to  a  Pilot  Study 


■ _ I 

2  I  Train  personnel  to  accomplish  change 

ll  '  1 

4  I  Evaluate  pilot  results  | 

5  j|  Hake  recor  liner  cl  afior  s  &  knpr  we  K§ 
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Evaluate  Pilot  Results 


Compile  performance  measurements  and  indicators.  Evaluate  how 
the  new  process  component  performed  with  respect  to  the  objectives 
and  success  criteria. 

Conduct  a  lessons-learned  meeting  with  pilot  study  personnel  to 
obtain 

•  feedback  on  the  new  solution  component;  what  worked  well 
and  what  didn’t  work  well 

•  ideas  for  improving  the  solution  component 

•  suggestions  for  improving  how  new  solution  components 
are  piloted 

In  addition,  consider  using  an  instrument  for  obtaining  anonymous 
feedback. 
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Statistical  Analyses  to  Validate  that 
the  Change  Worked 


Magnitude  of  improvement 

N - ►! 


Baseline  current  as-is  state  of 
practice 


The  new  state  of  practice 
after  improvement 
intervention 
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Are  You  New  to  Measurement 
Analysis? 

For  an  easy-to-understand  example  of  the  T-test  and 
Analysis  of  Covariance,  see: 

Kelley,  D.  Lynn  and  Morath  P.  How  Do  You  Know  The 
Change  Worked?.  Quality  Progress.  American 
Society  for  Quality,  pp.  68-74.  July  2001 . 

You  can  also  refer  to  a  statistical  textbook  or  reference 
book.  Statistical  software  packages  (e.g.,  SPSS, 
JMP/SAS)  are  also  capable  of  performing  these 
analyses. 
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Analyzing  the  Results 

Since  many  variables  may  contribute  to  the  pilot  results, 
it’s  important  not  to  draw  immediate  conclusions  without 
exploring  root  causes. 

What  factors  contributed  to  success  or  partial  success? 
What  factors  led  to  the  less-than-successful 
implementation? 

People  Process 
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A  Structured  Approach  to  a  Pilot  Study 


■ _ I 

2  I  Train  personnel  to  accomplish  change 

ll  '  1 

I _ I 

5  I  Make  recommendations  &  improvel 
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After  the  1 

Pilot  Study 

Outcome 

1 

Follow-on  steps 

Major  revision  *  p|an  the  revision 

required  *  Review  the  plan  with  pilot  project  personnel  and  get 

their  feedback — will  changes  address  concerns? 

*  Revise  the  solution  component 

•  Review  with  project  personnel — do  the  changes  address 
concerns? 


Conduct  another  pilot  study 


Minor  revision  *  Revise  the  solution  component 

Suggested  *  Review  with  project  personnel — do  the  changes  address 

concerns? 


Additional  support 
required  to  use 
solution 
component 


Plan  the  development  of  whole  product  support 
components  that  address  the  need 

Review  plan  with  pilot  project  personnel — will  additional 
support  address  concerns? 

Develop  additional  support  components 


Review  with  project  personnel — do  the  support 
components  address  concerns? 
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The  Value  of  Pilot  Feedback 


Feedback  from  the  pilot  can  help  you 


•  remove  bugs  from  the  solution  component  (i.e.,  the 
process,  procedure  or  technology) 


•  identify  ways  for  enhancing  the  solution  component 


identify  additional 
whole  product  components 
that  will  make  it  easier  for 
users  (in  the  actual 
implementation)  to  embraci 
the  process,  procedure  or 
technology 


The  Whole  Product  Wheel 
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Bringing  Closure  to  the  Pilot  Effort 

Communicate 

Meet  with  the  management  and  other  stakeholders  to 

•  review  pilot  results 

•  make  recommendations 

•  identify  next  steps 

Post  the  performance  results  of  the  pilot  effort  in  a  public  area 
for  review  from  the  organization. 
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The  Value  of  Conducting  Multiple  Pilot 
Studies 

Conducting  multiple  pilot  studies  leads  to  more  reliable 
information  for  decision-making. 

When  “testing”  a  new  technology,  investigators 
understand  that  replications  are  required  to  understand 
the  extent  of  experimental  error. 

Experimental  error  is  caused  by  the  variation  in  the  test 
results  caused  by  environmental  influences  that  are 
beyond  the  control  of  the  experimenter. 

To  mitigate  the  risk  of  experimental  error,  the  investigator 
repeats  the  experiment  multiple  times  to  better 
characterize  the  influence  of  the  technology  that  is  being 
tested. 
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Why  Did  We  Conduct  a  Pilot  Study? 

Piloting  reduces  the  risk  of  rolling  out  a  flawed  process, 
procedure  or  other  solution  component  to  a  broad  multi¬ 
project  environments. 

The  idea  behind  a  pilot  is  to  test  the  solution  component 
within  a  bounded  and  controlled  environment  before  the 
component  is  sanctioned  for  broader  use. 

During  a  pilot  study,  the  usability  of  the  solution 
component  is  evaluated  in  a  near  real-world  project 
setting. 

Experience  demonstrates  that  such  a  test  always 
exposes  improvement  opportunities  that  can  be  exploited 
to  hone  and  refine  the  solution  component  before 
broader  dissemination. 
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Contact  Information 


Telephone 

412/268-5800 

FAX 

412/268-5758 

Email 

customer-relations@sei.cmu.edu 

World  Wide  Web 

http://www.sei.cmu.edu 

U.S.  Mail 

Customer  Relations 

Software  Engineering  Institute 

Carnegie  Mellon  University 

Pittsburgh,  PA  15213-3890 

©  2003  by  Carnegie  Mellon  University  page  39 


